U.K.-based Deepmind has created artificial intelligence software that can read lips. An artificial intelligence (AI) program from DeepMind can read lips better than professional lip readers after reviewing thousands of hours of YouTube videos along with transcripts via machine learning. The researchers tested the program on 37 minutes of video it had not previously viewed, and it misidentified only 41% of the words. In comparison, the best previous computer method, which focuses on individual letters instead of phonemes, had a 77% word error rate, while professional lip readers had a 93% error rate in the same test, which lacked context or body language. Columbia University's Hassan Akbari says the AI, if incorporated into a phone, would enable hearing-impaired users to have a "translator" with them wherever they go.

For millions who can't hear, lip reading offers a window into conversations that would be lost without it. But the practice is hard--and the results are often inaccurate (as you can see in these Bad Lip Reading videos). Now, researchers are reporting a new artificial intelligence (AI) program that outperformed professional lip readers and the best AI to date, with just half the error rate of the previous best algorithm. If perfected and integrated into smart devices, the approach could put lip reading in the palm of everyone's hands. "It's a fantastic piece of work," says Helen Bear, a computer scientist at Queen Mary University of London who was not involved with the project.

When Nvidia popped the bonnet on its Co-Pilot "backseat driver" AI at this year's Consumer Electronics Show, most onlookers were struck by its ability to lip-read while tracking CES-going "motorists'" actions within the "car". A slide taken at CES shows the Co-Pilot AI assistant performing four features: facial recognition, head tracking, gaze tracking and lip-reading. The @nvidia AI co-pilot analyzes you through face recognition, head and gaze tracking and lip reading to assist you. The automative AI is part of the GPU-flinger's DRIVE PX 2 platform, which uses sensors and multiple neural networks powered by the grunt of Nvidia's processors. An Nvidia spokesperson has since confirmed in an email to The Register that the lip-reading component was based on research paper [PDF] written by academics from the University of Oxford, Google DeepMind and the Canadian Institute for Advanced Research.

Researchers from Google's UK-based artificial intelligence division DeepMind have collaborated with scientists from the University of Oxford to develop the world's most advanced lip-reading software – and it probably reads lips better than you. To accomplish this, the researchers fed thousands of hours of TV footage from the BBC to a neural network, training it to annotate videos based on mouth movement analysis with an accuracy of 46.8 percent. For context, when tasked with captioning the same video, a professional human lip-reader proved to be almost four times less efficient, accurately guessing the right word only 12.4 percent of the time. The research builds upon previously published work by the University of Oxford that used similar techniques to build a lip-reading app called LipNet that could read video recordings of volunteers speaking in simple sentences with an accuracy of over 90 percent. However, unlike Oxford's program, DeepMind's software – dubbed "Watch, Listen, Attend, and Spell" – was trained and tested on much more challenging footage.

A team of researchers at Oxford University have coaxed an artificial intelligence program into an impressive leap forward and towards our own obsolescence. The program, known as LipNet, is showing particularly promising ability to read lips in video clips, thanks to machine learning and a novel way of approaching the data. The key difference is that rather than try to teach the AI the mouth shapes of single words and phonemes, the LipNet is asked to interpret whole sentences. Using GRID, a huge bank of 3 second videos featuring brightly lit forward facing speakers, LipNet has learned to translate speech to text with a 93.4% accuracy rate. Compare that to humans' 52.3%.

Lip reading is a way of understanding speech by interpreting a person's lip movement. However, human speech is highly complex and nuanced, where one lip movement could correspond to different phonemes, or basic units of sound. Therefore, the practice is prone to errors, which can sometimes lead to humorous results. Scientists from Oxford University have described an artificial intelligence system, called LipNet, which can accurately read lips. The system employs deep learning to train itself using 29,000 three-second-long videos labeled with captions.

Lip reading, an essential tool that helps the hearing-impaired to better understand the world, is now conducted by artificial intelligence with a better accuracy than done by humans, University of Oxford reveals. In an article currently published by Quartz we learn that a new paper issued by the University of Oxford with funding from Alphabet's Deepmind, reveals that they have developed an artificial intelligence system called LipNet that can read lips with an accuracy of 93.4%. University of Oxford has previously released a system that operated word-by-word with an accuracy of 79.9%, but their new system has now developed a new and different way of approaching the problem. "Instead of teaching the AI each mouth movement using a system of visual phonemes, they built it to process whole at a time. That allowed the AI to teach itself what letter corresponds to each slight mouth movement", Quartz writes. The new system was exposed to 29 000 3-second-videos videos labelled with the correct text to train the system, and in comparison with human lip-readers that had an average error rate of 47.7%, the AI's error rate was only 6.6%.

The researchers, working with Google's artificial intelligence division DeepMind, trained the software on more than 30,000 videos of test subjects speaking sentences. Over time, it would match certain words with particular lip movements to learn what words were being spoken. The researchers then played it further videos of people speaking sentences and the LipNet software succeeded with 93.4 per cent accuracy. This compares to 52.3 per cent for hearing impaired students, and surpassed other lip-reading programs. Unlike previous software, LipNet digested the phrases as full sentences, and allowing it to put words in context rather than decipher them individually allowed much greater accuracy.

Even professional lip-readers can figure out only 20% to 60% of what a person is saying. Slight movements of a person's lips at the speed of natural speech are immensely difficult to reliably understand, especially from a distance or if the lips are obscured. And lip-reading isn't just a plot point in NCIS: It's an essential tool to understand the world for the hearing-impaired, and if automated reliably, could help millions. A new paper (pdf) from the University of Oxford (with funding from Alphabet's DeepMind) details an artificial intelligence system, called LipNet, that watches video of a person speaking and matches text to the movement of their mouth with 93.4% accuracy. The previous state of the art system operated word-by-word, and had an accuracy of 79.6%.

Natasha Lomas reports in TechCrunch, "For human lip readers, context is key in deciphering words stripped of the full nuance of their audio cues. But a technology model for lip-reading developed at the University of East Anglia in the UK has been shown to be able to interpret mouthed words with a greater degree of accuracy than human lip readers, thanks to the application of machine learning tech to classify the visual aspect of sounds. And the kicker is the algorithm doesn't need to know the context of what you're discussing to be able to identify the words you're using." Lomas goes on, "While the model remains a piece of research at this stage, there are scores of potential applications for technology that could automagically transform visual cues into accurate speech -- whether it's helping people who have audio impairments, or enhancing audio-less security video footage with additional speech data -- or even to try to figure out exactly what charged word one footballer spat at another in the heat of a match." She continues, "Such a tech could also be applied as a fallback for poor audio quality on a mobile or video call. Or even perhaps to power a front-facing camera-based mobile'voice' assistant which you wouldn't actually have to speak to but could just discreetly mouth commands at (how cool would that be?). Safe to say, the list of applications-in-waiting for machine powered lip-reading is as long as the dictionary is deep. So there's bags of future potential if only researchers can deliver the goods."